Query processing for datacenter-scale computers

Quickly exploring massive datasets for insights requires an efficient data processing platform. Parallel database management systems were originally designed to scale only to a handful of nodes, where each node keeps recent (“hot”) data in memory and has directlyattached hard disk storage for infrequently accessed (“cold”) data. To keep pace with the growing data volumes, the research focus has been shifting towards rack-scale architectures. Rack-scale systems combine powerful nodes with directly-attached storage for “warm” data with hundreds of terabytes of network-attached storage for “cold” data. Exemplars include Oracle Exadata, the IBM PureData System and the Microsoft Analytics Platform System. Processing even larger datasets quickly will inevitably require datacenter-scale computers. Although details of the hardware configurations of commercial datacenters are scarce, one can use scientific computers for data-intensive applications as a proxy. Compared to rack-scale architectures, the “hot” data path consists of a few petabytes of memory fragmented across ten thousand nodes, and proprietary network interconnects in unique topologies. The “cold” data path retrieves data through a parallel file system (such as Lustre) from many petabytes of network-attached storage. We posit that the optimizations a DBMS performs individually within a node or across a handful of powerful nodes are insufficient when query processing becomes a datacenter-scale challenge. We argue that the opportunity for database systems research is to lead the transition from the cloud-inspired view of a datacenter as a collection of CPUs towards the view of a datacenter as a single entity that requires careful task orchestration for peak aggregate performance. We identify the following research directions towards realizing this vision.